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This paper examines the complexities of working with student researchers in a virtual lab 
setting, logistics, and methods to resolve issues. To demonstrate the feasibility of a virtual 
lab, a mixed-methods study consisting of quantitative surveys and qualitative data examined 
changes in doctoral students’ confidence as measured by research outcome expectations and 
changes in a self-assessment of research knowledge and skills test in a 3-quarter virtual 
psychology research lab. In the lab, 10 doctoral students conducted a faculty-designed 
project, analyzed data, and cowrote a literature review. Findings indicate lab students’ 
research knowledge and research outcome expectations improved significantly over nonlab 
comparison students. Student journals provided qualitative evidence of the student 
experience in a virtual research lab. Students reported individual growth, self-learning, and 
appreciation of the shared group experience with a common goal. Students related that the 
application of skills learned in classes to a research setting was more difficult than 
anticipated. 
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Introduction 

A key element of graduate training in the social sciences is in-depth intellectual mentoring, 
particularly through research training. Research mentoring benefits students through the 
development of professional skills, increased confidence and self-efficacy (Love, Bahner, Jones, & 
Nilsson, 2007), dissertation success, reported satisfaction of the doctoral program in which the 
student was enrolled (Clark, Harden, & Johnson, 2000), and improved career potential (Demaray, 
Carlson, & Hodgson, 2003). 

In contrast to undergraduate education, where learning occurs primarily in the classroom, the 
production of scholars at the graduate level occurs primarily through the process of mentoring 
(Forehand, 2008). There has been concern as to the ability of faculty to mentor students in an online 
environment (Belar, 2006) for the same reasons online faculty have had difficulty in conducting 
research: a lack of laboratories and student opportunities to assist in research. Currently, there is 
little opportunity for students to assist in faculty research projects online, due to the difficulty in 
conducting research with students in a virtual environment. Of particular concern to potential 
faculty supervisors is the ability to supervise lab students, maintain data confidentiality, and 
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maintain the integrity of the study protocol. There is a paucity of research or information regarding 
how to do research with student researchers in an online research setting, which deprives online 
faculty of an understanding as to how to conduct such a research project and provide students with 
an experience equivalent to that in a land-based program. 

In a land-based lab, students gain exposure to a positive research-training environment, faculty 
modeling of research skills and enthusiasm for the research process, and the social experience of 
working with others (Love et al., 2007). These factors have been reported to increase student 
research self-efficacy and lead to higher research productivity (Hollingsworth & Fassinger, 2002; 
Phillips & Russell, 1994). The land-based academic research training environment has been found to 
predict research outcome expectations (Bishop & Bieschke, 1998; Kahn, 2001) and research 
investigative interests (Bishop & Bieschke, 1998; Kahn, 2001). 

It is not known, however, if a virtual lab can provide the research skills and research interest as seen 
in a land-based lab. In an attempt to examine this issue, the current study recruited 10 doctoral 
psychology students at an online university to participate in a 3-quarter virtual lab, in which they 
assisted in a faculty-designed research project. A comparison group of students was matched with 
the lab students based upon milestones in program progress (e.g., completion of all required courses). 
A measure of confidence in the development of research skills (measured with Bieschke and Bishop's 
[1994] Research Outcome Expectations scale), and a researcher designed research knowledge and 
skills self-assessment examined the students' experience, as well as weekly journals from the 3 
quarters in the lab. In order to place the current virtual lab study into context, a brief history of labs 
in academia will be examined, as well as what is known about virtual research labs. 

Laboratories in Academia 

The use of seminar laboratories in an academic environment began in the 19th century (McNeely, 
2009) and developed into an apprenticeship-type method to train students in conducting research. By 
the early 1900s, laboratories moved from being a domain of the natural sciences to being used by the 
social sciences whereby education, psychology, and public health began to be studied in an 
experimental setting. The 1940s through 1960s were a period of growth in higher education and 
resulted in a subsequent increase in research due to the increased funding for student tuition 
available through the Servicemen's Readjustment Act of 1944 (also known as the G.I. Bill of Rights; 
Walker, Golde, Jones, Bueschel, & Hutchings, 2008). During the 1960s, an increase in college 
enrollment from the Baby Boomers combined with an increase in federal research dollars resulted in 
a link between faculty research agendas and graduate student training in research. Graduate faculty 
teaching loads were reduced, allowing more time for laboratory research in the sciences, social 
sciences, and humanities (Walker et al., 2008). 

Today, the laboratory is considered essential to many social science disciplines as an environment to 
study aspects of human behavior and as a mechanism to train students in the application of skills 
learned in the classroom. Typically, the laboratory serves a dual role from the faculty perspective: (a) 
it is a way to collect the data necessary to publish and to secure employment as a professor in the 
university and to be recognized as an expert in the field of study, and (b) it is a way to mentor 
students through the steps of a research project and teach them the skills necessary to conduct their 
own projects for their thesis and dissertation work and to become professionals in the field. An added 
benefit for many students is the potential for being coauthors on professional journal articles, which 
today is becoming a requirement for beginning faculty positions at research universities. 
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Depending upon the institution, some research mentoring may be done on an undergraduate level, 
often just with honor students. Research mentoring is expected to occur on the graduate level, and 
the importance of research mentoring in graduate education has long been a concern. Before the 
1960s, a research-focused model leading to the Ph.D. was typical (Ellis, 1992; Roe, Gustad, & Moore, 
1959). During this time, students were trained to teach and do research with the expectation that 
they would enter into academic positions (Altman, 1990; Ellis, 1992). In the 1960s, as more applied 
career options became available across disciplines, the traditional model began to be questioned 
(Ellis, 1992). The issue of research mentoring remains contentious, particularly for applied 
disciplines (Gelso, 1993, 2006); however, in programs leading to a Ph.D., research mentoring appears 
to be an inevitable need. 

Moving Academia Online 

Distance education derives from correspondence courses, first initiated in 1852 (Casey, 2008). The 
first course was the Pitman Shorthand training program that brought innovative stenographic 
practices to the United States. Using the United States Postal Service, self-taught secretaries would 
mail their exercises to the Phonographic Institute in Cincinnati, Ohio, and after completing the 
required coursework, received a certificate of expertise in stenographic shorthand skills (Casey, 

2008; Matthews, 1999). With the advent of the computer, online programs were first developed in 
1993 at Jones International University in Boulder, Colorado. The university provided instruction in 
five bachelor's degrees and 24 master's degree programs and was the first fully online university to 
be accredited by the Higher Learning Commission (Casey, 2008). 

The essential characteristic of distance learning programs is that students and instructors are 
geographically dispersed (Murphy, Levant, Hall, & Glueckauf, 2007). Students and instructors may 
be not only anywhere in the United States, but also throughout the world. With the advent of the 
Internet, distance learning programs can incorporate video, email, and various classroom software 
(e.g., Blackboard, eCampus), as well as telephone communication and in-person meetings. 

Today, distance learning has become a ubiquitous entity: more than 6 million students participated 
in at least one online course in the fall of 2010, an increase of more than half a million students over 
the previous year (Allen & Seaman, 2011). Between fall 2007 and fall 2008, the online enrollment 
growth rate increased 12%, while the overall higher education enrollment growth rates increased 
only 1.2% (Allen & Seaman, 2008). 

Virtual Labs 

There is very little literature on virtual or online research laboratories in graduate education, with 
existing research typically discussing the use of shared remote equipment. For example, Gonzalez- 
Castano and colleagues (2001) designed an Internet access laboratory that provided remote access to 
equipment used in a computer architecture laboratory. There were no descriptions of methodological- 
based virtual research labs in the literature until Stadtlander and Giles (2010), however. 

Stadtlander and Giles (2010) discussed a virtual qualitative-method lab in psychology in which 
students assisted in collecting data from 35 elderly participants across the United States. Each of 
the seven graduate students in the lab conducted 3-hour interviews with their participants. Weekly 
student journals indicated that the students enjoyed the lab and felt they learned a great deal about 
the research process through the experience; however, the authors expressed concern as to whether 
the students had gained competence, research knowledge, and skills beyond their regular program of 
study. 
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There is a body of literature on collaboratories, e-Science, and cyberinfrastructure that relates to the 
current conception of the virtual lab. A collaboratory has been defined as “center(s) without walls in 
which researchers can work together regardless of physical location” (Wulf, 1993, p. 854). This 
concept has been used by professional researchers in a number of ways, such as email or a virtual 
space in which they collaborate on projects, generally in the physical or biological sciences and 
engineering. The terms e-Science (used in Europe) and cyberstructure (seen in the United States; 
Atkins et al., 2003) were developed to refer to the same or related ideas as the word collaboratories, 
but generally on a larger scale (Olson, Bos, & Zimmerman, 2008). They are all structured, however, 
to be collaborative spaces for professional or faculty researchers—not as a means to educate students 
in the research process, which is the basis for the current virtual lab. 

The Lab: Virtual Versus Land-Based 

The virtual lab environment offers challenges not present in a land-based lab. In a land-based lab, 
students have the advantage of being more closely supervised than is possible online. It is imperative 
that students working in a virtual lab be trained sufficiently so that they can handle any issues that 
arise independently. This means that they must have a good understanding of the methodology and 
have carefully considered a plan of action for problems. For example, the present project examined 
people in the oldest-old age group (over 85). Students must be prepared for issues that may arise, 
such as evidence of elder abuse and suspected dementia, which excludes the individual from 
inclusion in the study, as well as how to address issues with staff in facilities. In a virtual lab, 
students are often more coresearchers than in a land-based lab. 

An additional difference between a virtual versus land-based lab is the issue of data ownership and 
security. In a land-based lab, it is relatively easy for the faculty to maintain control of the data and 
to provide security by locking the lab computer or door to the physical lab. Online, the students have 
continued access to the data that they have collected (and perhaps other students' data). It must be 
clearly and repeatedly stated who has ownership of the data and what, if anything, the students can 
do with the data independently. 

Another element is training the students in the research protocol, which tends to be more complex 
online than in an in-person environment. In the current project's study, students were trained to 
conduct interviews. Training consisted of readings, an example interview between instructors, and 
three recorded practice interviews with pilot subjects. All student contact with pilot and study 
participants was recorded and reviewed by the faculty. 

A security plan also must be in place to make sure that the confidentiality of the study participants 
is not violated. In the authors' virtual lab, students were required to send all data, including 
recording files through an encrypted website (YouSendIt.com). Students purchased a flash drive and 
maintained all data on the drive with password protection on the files. 

Research Outcome Expectations 

Research outcome expectations have been identified as accounting for more variance in students' 
research interests than any other significant variable, including research self-efficacy beliefs (Bard, 
Bieschke, Herbert, & Eberz, 2000; Bishop & Bieschke, 1998), the research training environment 
(Bishop & Bieschke, 1998; Kahn, 2001), investigative interests (Kahn, 2001), and age (Bishop & 
Bieschke, 1998). Kahn has speculated that students may be unable to discriminate between their 
interest in research and their expectations about the outcomes that may result from engaging in 
research activities. In the present study, the Research Outcome Expectations Questionnaire 
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(Bieschke & Bishop, 1994) was used as a measure of student researchers' confidence in the 
development of research competencies. 

Research Knowledge and Skills 

Bieschke (2006; also see Gelso & Lent, 2000; Heppner, Kivlighan, & Wampold, 1999) stressed the 
importance of examining actual competence as a researcher. Focusing on the precursors (e.g., 
interest in research) or the outcomes of competencies (e.g., research productivity) rather than the 
competencies themselves seems to overlook a valuable aspect of research training; however, there 
are no measures available in the literature for examining student research competencies. A measure 
of research knowledge and skills was developed by the researchers (see Appendix). The 34 items in 
the measure were selected from learning outcomes in two research methods and two statistics 
classes that all students were required to have taken in order to be in the study. Next, the measure 
was piloted with 11 dissertation students, the students' dissertation advisors were interviewed as to 
their qualitative assessments of the students' skill levels in each area, and the results were 
compared. The present study allowed the measure to be piloted over a 3-quarter period to examine 
reliability over time (reliability statistics will be presented in the Method section). Lab students who 
participated in the project for 3 quarters were matched with comparison students who continued in 
their program study without the lab. Both groups of students took the same measures at the same 
four points in time. 

There were four main research questions in the current study: 

1. Is there a difference in lab students' confidence in the development of research competency 
(as measure by the Research Outcome Expectations Questionnaire [Bieschke & Bishop, 

1994]) over the 3 quarters of the virtual lab as compared to a group of matched comparison 
students? 

2. Is the researcher-designed student self-assessment of research knowledge and skills a 
reliable measure? 

3. Is there a difference in lab students' self-assessed research knowledge and skills over the 3 
quarters of the virtual lab as compared to a group of matched comparison students? 

4. How do the lab students qualitatively experience the 3 quarters of the virtual lab? 

The Lab Course and the Current Study 

In 2011, the first author received a grant from Walden University to conduct a mixed-methods study 
(consisting of quantitative surveys and qualitative interviews) on the current health experiences of 
people over 85 years (the oldest-old). In order to have the opportunity to gain elderly participants 
from across the United States, allow student participation, and be able to study students working in 
a virtual lab, the authors developed a virtual laboratory. eCollege software provided the framework, 
which allowed for asynchronous discussions and the ability to post information, such as downloaded 
Word files. Three faculty members supervised the project; all of them responded in the class and 
participated in twice-monthly lab meetings with the students. The course, Independent Research, 
was an elective course available to doctoral students in the clinical, health, or general psychology 
specialties. 

Walden University student listservs posted the availability of the project, with interested students 
completing applications. Twenty-nine individuals submitted an application, which were then sorted 
by area of the country and priority was given to those students who had completed research methods 
courses and had an interest in gerontology. This resulted in 10 students selected and registered in 
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the course. Ten additional psychology students were chosen from the qualified applicants (matched 
with lab students to be at similar milestone in their doctoral program, e.g., writing dissertation) to 
be comparisons for the study. 

Students were asked to commit to working on the project for 3 quarters (9 months) and lab students 
agreed to each conduct four 1-hour interviews (not all students were able to recruit the four 
participants) and administer quantitative surveys. All lab and comparison students took the 
Research Outcome Expectations Questionnaire and Research Knowledge and Skills Self-Assessment 
at the beginning of the project and at the end of each quarter to evaluate learning and research 
expectations over the entire 3-quarter period. Lab students also submitted a brief reflection journal 
each week, discussing their impressions and opinions on the virtual lab experience. 

The mixed-methods project that the students worked on during the 3-quarter lab was a continuation 
of Stadtlander and Giles’ (2008) qualitative study on the elderly. The study examined oldest-old 
participants' interaction with their physicians through quantitative measures and an interview. The 
students conducted a 1-hour qualitative interview with each of their participants in which the 
elderly individuals examined their thoughts and opinions on their healthcare and physicians. The 
quantitative portion included a number of survey instruments to examine participants’ current 
health, healthcare history and health efficacy. 

Prior to the first quarter, all lab students received an Olympus digital voice recorder, purchased 
through the grant. The instructors developed and tested interview questions in focus groups and 
selected quantitative measures for the study that the students would conduct. The instructors 
developed consent forms, demographic questions, and scripts for contacting participants, which were 
all sent to the students. They also submitted an application to Walden University’s Institutional 
Review Board for both the present study on lab versus comparison students and the study that the 
students conducted. 

During the first quarter, lab students were trained on the methodology and protocol, and each 
student conducted interviews and surveys with four elderly individuals. During the second quarter, 
students read articles and texts on mixed-methods analyses and conducted both quantitative and 
qualitative analyses on the collected data. In the third quarter, students had readings on technical 
writing and jointly wrote a literature review on the project. 

Students and instructors participated in lab meetings every two weeks during the 3 quarters using 
GoToMeeting software. During the meetings, the instructors provided additional training, answered 
questions, and clarified procedures. The instructors recorded the meetings and posted the recordings 
in the classroom for later review, as needed. 


Method 


Participant Demographics 

Student applicants were recruited through Walden student listservs, and were asked to complete an 
application. Ten qualified psychology students were chosen as lab students based upon an interest in 
gerontology and their geographical location to ensure a nationally representative sample for the 
primary elderly study. The students age ranged from 35 to 67 years, M = 49.2 years; all were female, 
eight self-described as Caucasian, one biracial, and one as African American. 
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A possible confound in the study was that students may increase their research knowledge and 
research confidence through their education without the lab opportunity; therefore, a comparison 
group was used. Ten additional psychology students were chosen from the qualified applicants 
(matched with lab students to be at a similar milestone in their doctoral program, e.g., writing 
dissertation) to be comparisons for the study. Their age ranged from 40 to 67, M = 49.2 years; nine 
were female and one male, seven self-described as Caucasian, one Middle Eastern, and one as 
African American. 

All 20 students (10 lab students and 10 comparisons) had taken at least two previous graduate-level 
research methods and two statistics courses. All participants took the two measures at the same four 
times: at the beginning of the project and at the end of each of the subsequent 3 quarters. The 
comparison students received a $50 gift card to Amazon.com in appreciation for their participation. 
Lab students participated in a virtual classroom in which they received mentoring and had twice- 
monthly conference calls with the instructors. Comparison students received an email at the end of 
each quarter to complete the survey measures. 

Quantitative Measures 

The Research Outcome Expectations Questionnaire was used (Bieschke, 2006; Bieschke, 2000; 
Bieschke & Bishop, 1994). This scale consists of 17 items reflecting both professional and personal 
positive outcomes (e.g., "My involvement in research will lead to meaningful contributions to the 
field") and one item reflecting negative outcomes (e.g., "Involvement in research will take time away 
from leisure activities") that might result from participation in research activities. Participants are 
instructed to indicate their degree of agreement with each statement using a 5-point Likert-type 
scale ranging from 1 (strongly disagree) to 5 (strongly agree). Reliability data have revealed good 
internal consistency with the scale, with coefficient alphas of 0.89 (Bieschke & Bishop, 1994; 1995) 
and 0.90 (Bieschke, Bishop, & Herbert, 1995). In the present study, internal consistency estimates 
using coefficient alpha was 0.82 for the pretest, 0.83 at the end of quarter 1, 0.84 at the end of 
quarter 2, and 0.80 at the end of quarter 3. 

A Research Knowledge and Skills Self-Assessment measure was developed and piloted in the current 
study by the authors; the measure consisted of 34 research skills necessary in the study. The skills 
were selected from learning outcomes in two research methods and two statistics classes that all 
students were required to have taken in order to be in the study. Example skills included, "Identify 
gaps and patterns in existing research and generate researchable ideas" and "State the legal and 
ethical considerations associated with qualitative research." Participants rated their competence 
with each skill using a 5-point Likert scale ranging from 1 (I do not feel strong in this area - beginner 
level) to 5 (I feel I am extremely strong in this area - expert level). In the present study, internal 
consistency estimates using coefficient alpha shown to be reliable: 0.92 for the pretest and 0.97 at 
the end of the next three measurement periods. In order to examine the measure's stability over time 
(i.e., test-retest reliability), three sets of time-related (T1-T4) correlations were conducted. Three 
proximal test-retest correlations (T1 with T2 [r = .88, p < .01], T2 with T3 [r = .71, p < .01], T3 with 
T4 [r = .80, p < .01]), two intermediate (T1 with T3 [r = .50, p < .05], T2 with T4 [r = .46, p = .056]), 
and one distal (T1 with T4 [r = .29, p > .05]) were conducted. These findings suggest that the 
measure is stable proximally at a 3-month period, somewhat stable at a 6-month period, and does 
not show stability over a 9-month period. 
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Qualitative Data 

In order to understand the virtual lab students' shared experience regarding the learning and skill 
development in the virtual lab, students wrote a one-page journal entry and submitted it weekly. In 
this assignment, they received the following instructions: 

Talk about how you are feeling about the training you are being given. Does 
it feel like enough training in its relevance to the project? How do you feel 
about the doing the project, and do you feel like there enough supervision? Do 
you have ideas on how things could be improved? 

There was variation in how students approached journaling—some were insightful and self- 
revealing while others tended to recite statements that the individual had shared with the lab 
earlier in the week through discussions. Not all students were timely in submitting the journals; late 
journals were not used in the qualitative analysis. This qualitative data was examined to find 
themes that clarified the learning experience from the student perspective. Identified themes further 
examined how students' learning and skill development progressed from a personal perspective. 

Analyses 

In order to qualitatively analyze the lab students’ journals, week 6 and 12 journals were examined 
for each of the three terms. Only journal entries that addressed at least one of the following topics 
were included: comments on the study, personal insights/thoughts, and comments on group 
interaction. No late assignments were used. Some students reflected on personal issues in their lives; 
these were not included in the analysis. The responses were coded and themes explored. 

Ethical Issues 

Both the students’ elderly project and the current lab study received Walden University Institutional 
Review Board approval (approval #03-09-11-0784474). All students signed informed consent forms 
before they committed to participate in the study, as did individuals in the elderly study. The 
transcriptionist and student researchers signed confidentiality agreements. 

Results 


Quantitative Data 

For the Research Knowledge and Skills Self-Assessment, a repeated-measures ANOVA was 
conducted for the total score for each of the four time periods. As shown in Table 1, there was a 
significant main effect of time (.F[l, 14] = 20.06, p < .001), thus, the overall means for each of the four 
time periods were significantly different. Post-hoc t-tests indicated the pretest differed from all later 
tests ip < .01); the end of quarter 1 test differed from the two later tests (p < .01). There was not a 
main effect of condition (p > .05). 


Table 1: Research Knowledge Test Scores for Lab and Comparison Students (SE in Parentheses) 



Pretest 

End Quarter 1 

End Quarter 2 

End Quarter 3 

Means 

Lab group 

91.3 (6.9) 

104.9 (6.7) 

132.1 (7.6) 

137.4 (6.1) 

116.4 

Comparison group 

119.4 (6.9) 

123.8 (6.5) 

125.1 (8.1) 

124.6 (9.3) 

123.2 

Means 

105.3 

114.3 

128.6 

131.0 
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As shown in Figure 1, there was an interaction effect (F[l, 14] = 13.25, p < .01), whereby the lab 
group showed a greater knowledge gain over the four periods as compared to the comparison group. 

160 
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20 
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Pretest End term 1 End term 2 End term 3 

Figure 1: Research Knowledge Test Scores Interaction Effect for Lab and Comparison Students 



— Lab group 

Control group 


For the Research Outcome Expectations Questionnaire, a repeated measures ANOVA was conducted 
for the total score across all four time periods. As shown in Table 2, there was a significant main 
effect of time (F[l, 14] = 10.2, p < .01), thus the overall means for each of the four time periods were 
significantly different. Post hoc i-tests indicated that the pretest differed from the test at the end of 
quarter 2 and quarter 3 (p < .05); the test at the end of quarter 1 differed from the tests at the end of 
terms 2 and 3 (p < .05). There was also an effect of condition (F[l, 14] = 6.3, p < .05) in which the 
overall mean for the lab students was higher than the comparison group. There was no interaction 
effect (p > .10). 


Table 2: Research Outcome Expectations Questionnaire Scores for Lab and Comparison Students 


(SE in Parentheses) 



Pretest 

End Quarter 1 

End Quarter 2 

End Quarter 3 

Means 

Lab group 

71.3 (1.6) 

73.6 (1.6) 

76.38 (1.2) 

76.88 (1.2) 

73.78 

Comparison group 

69.1 (1.9) 

69.4 (1.9) 

68.63 (1.8) 

70.25 (1.1) 

68.47 

Means 

68.4 

70.1 

72.5 

73.6 



There was no relationship between scores on the Research Outcomes Expectations Questionnaire 
and the Research Knowledge and Skills Self-Assessment at any of the four testing times (ps > .05). 
This suggests that the two measures examined different elements of the research process. 
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Qualitative Data 

The qualitative data was coded and analyzed for both overarching themes and themes that tied 
directly to the specific focus of the virtual lab in each of the three terms. The themes that emerged 
emphasized aspects beyond those measured by the quantitative measures that affected the student 
experience. The themes clarified the personal aspects that most influenced the students' 
development of Research Outcome Expectations Questionnaire and knowledge. 

Overarching Themes (Themes Over Three Terms) 

Students identified their individual growth and self-learning, especially of their own self-limitations. 
They realized that they did not know how to apply concepts and skills they had previously learned in 
other courses. Students further identified that they were not only able to identify problems within 
their skill sets, but were able to develop plans to solve these problems. 

I suppose that the learning that occurs in the [other] classes is more on a 
knowledge base and increasing that base. The exposure I have had through 
the project is on a number of different levels. The actual immersion in the 
process, completing portions of the process and then putting the portions 
together encompasses other levels. (Student D) 

The final overarching theme seen was the student appreciation of group connectedness. 

Quarter 1 Themes 

During this phase of the project, students learned how to interview, contact subjects, administer 
informed consent, collect data, and many other tasks involved in data collection. The themes that 
emerged from the journals during this phase related to the learning process. The students reflected 
on learning new skills, gaining knowledge about the elements involved in research, and gaining a 
clearer understanding of the reality of conducting a research project. In terms of the reality of 
research, the themes that emerged were time management, flexibility, the need for group support, 
and the appreciation of the shared experiences. Throughout this quarter, the students reflected 
excitement, confidence, and expanding and learning new skills. "This journey continues to prompt 
me to learn about myself and the experiences I have. I continue to expand my levels of knowledge." 
(Student D) 

Quarter 2 Themes 

The second quarter was designed to be primarily analysis of the data collected in the first quarter. 
Themes that emerged from student journals again focused around learning. There was an awareness 
of a lack of knowledge, but also an enjoyment of the research process. Students identified the mental 
challenge of this phase, but also reflected on the development and application of new skills, decision¬ 
making, and accomplishment. 

I will honestly say that before this point, I had not really thought a lot about 
the interview answers that some of my participants provided. Going back and 
transcribing them made me really think about the answers and what they 
had in common. (Student C) 

A final theme that emerged during this phase of the project was the group learning process and the 
need and appreciation of regular conference calls with the entire lab. 
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Quarter 3 Themes 

The final quarter of the virtual lab was designed to have students do a complete literature review on 
the project, and begin a journal quality write up of the research. Again, learning was a strong theme 
throughout the student experience. Throughout the theme of learning, the areas identified varied. 
Students identified, as in previous terms, that they learned skills (e.g., writing, APA format, 
literature searches) but also identified learning in terms of the self: attempting perfection, reducing 
anxiety, learning to ask questions, assisting in the dissertation process. They also became more 
focused on themes regarding an appreciation for the elderly, an awareness of the elderly, and a 
clearer understanding of issues facing the oldest-old. The students also identified the need and 
appreciation of regular conference calls again in this quarter. It was in this quarter that the theme of 
flexibility reemerged but was linked with a theme of fatigue in the process. 

I would like to think that my awareness of issues experienced by those 85 
and older has increased. I find myself more conscious of those individuals I 
encounter on a daily basis. I feel that I have been able to increase this 
awareness in others as well. (Student D) 

My steam is beginning to run out, which is starting to be reflective in my 
work quality. I have to get over that and keep moving :>). (Student E) 

The themes that emerged over the three terms appeared to indicate that although much of the 
learning was skill-based, learning was equally valued by the students in terms of the application of 
knowledge that had been gained previously in the classroom to a real research project. It was also 
clear that peer support and group communication enhanced learning. 

Discussion 

Involvement in the virtual lab significantly improved the students' research knowledge and skills 
and student researchers' confidence in the development of research competencies (as measured by 
the Research Outcome Expectations Questionnaire [Bieschke & Bishop, 1994]) over comparison 
students matched to be at the same points in their doctoral program. These findings are consistent 
with previous research in land-based labs (e.g., Bard et al., 2000; Bishop & Brieschke, 1998). 

There has not previously been a measure of student researchers' knowledge and skills available. The 
current pilot of this measure suggests that the survey appears to be reliable and stable up to a 6- 
month period; it appears that it may be worth further exploration and validation with a larger 
sample. While the increase in lab students' research knowledge may appear to demonstrate simply 
an advantage of having additional experiences, it is also important to note that such applied 
research experience has been lacking in online graduate education. The comparison students 
participated in the normal graduate program and subsequently lacked the gain in confidence and 
knowledge that was seen in the lab students, thus demonstrating the advantage of having online 
students participate in an applied research project. 

Students showed individual growth and self-learning. Many reported appreciating the shared 
experience of working and learning in a group with a common goal. Repeatedly, students reported 
that while they had studied and applied concepts and skills in a classroom setting, the application of 
those skills in a research setting was more complex. For example, everyone had taken statistics 
courses, but the application with a real data set was more difficult than expected for the students. 
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This study has demonstrated that it is possible for online faculty to provide research experience to 
students in a virtual environment that is comparable to that in a land-based setting. A measure of 
student research knowledge and skills was piloted and appears to be reliable over time; further 
validation of the measure is warranted. There are a number of logistical issues that faculty must 
address in an online research environment with student researchers; a summary of these issues is 
provided for faculty who may wish to develop their own virtual lab. 

Logistical Issues 

There are challenges specific to teaching a virtual lab. For example, training students with 
methodology is a challenge; monitoring their protocol compliance requires extensive communication 
and considerable trial and error. Each student’s skills need to be evaluated and remediated online. 
Similar to land-based labs, personality issues can arise between students and between student and 
faculty, which must be carefully and judiciously handled with the added complication of the online 
environment. 

The lab used a stripped course shell, which required that the instructor(s) develop their own syllabus 
and set up the course and grade book. We have found that having multiple instructors (there were 
three in the current project) is essential, particularly when reviewing recordings for a qualitative, 
mixed-methods project. We designated an individual faculty member to be in charge of each aspect of 
the project (e.g., monitoring ethical compliance, maintaining data and analyses), and all faculty 
members participated in the classroom instruction; one person was designated to assign grades. 
Faculty members need administration support through course buyout, workload adjustments, and 
tech support. We have found that supportive technology is useful, such as in using GoToMeeting 
software, maintaining a blog, and using YouSendIt.com for large data and recording files. 

It is important to be clear to students on expectations, ownership of data, and potential author 
status on papers (we included this information in the student application and in the syllabi). 
Instructors should be mentors to the lab students by having regular lab meetings, assessing 
students' skills, and providing necessary training. It is also important for instructors to be flexible in 
both their own and students' expectations and be mindful of group dynamics: strong student 
personalities may have a negative impact on the research and class interactions. 

We have demonstrated that faculty mentoring in a virtual lab environment effectively improves 
student research knowledge and skills and increases student research confidence while providing 
faculty the opportunity to collect national data in a short time period. The logistics of such a lab can 
be daunting, but the opportunities provided for the faculty and students are worth the difficulties. 
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Appendix 

Research Knowledge Questionnaire 

There are no right answers on this questionnaire; it is simply a mechanism for us to look at areas 
that we might help you. Please try to accurately assess your own ability on each of the following 
items on a scale of 1 (I do not feel strong in this area - beginner level) to 5 (7 feel I am extremely strong 
in this area - expert level). You can bold your answer or change the font to a different color, if you 
wish. 



I do not 
feel 

strong in 

this area 
(beginner) 

I feel I 
have little 
strength 

in this 

area 

I have 

marginal 

or 

borderline 
strength in 

this area 

I feel I am 

reasonably 
strong in 

this area 

I feel I am 
extremely 
strong in 

this area 
(expert) 

Obtain certification in 

Protection of Human 

Research Subjects (NIH) 

1 

2 

3 

4 

5 

Anticipate ethical concerns 
related to conducting 
research with human 
participants 

1 

2 

3 

4 

5 

Begin a literature review for an 
area of interest 

1 

2 

3 

4 

5 

Analyze, evaluate, and 

synthesize research articles 

1 

2 

3 

4 

5 

Identify gaps and patterns in 
existing research and 
generate researchable ideas 

1 

2 

3 

4 

5 

Differentiate between types of 
research approaches and 
understand the rationale of 
each approach 

1 

2 

3 

4 

5 

Understand the strengths and 
limitations of different 
research 

approaches/methods 

1 

2 

3 

4 

5 

Assess the strengths and 
limitations of quantitative 
research designs 

1 

2 

3 

4 

5 

Choose the appropriate 

population and sample for a 

1 

2 

3 

4 

5 
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research design 


Critically evaluate qualitative 1 2 

research methods 

State the legal and ethical 1 2 

considerations associated 
with qualitative research 

Prepare for and perform 1 2 

observations and interview 

Assess the roles of bias, context, 1 2 


and the researcher in 
qualitative research 

Describe and discuss the 1 2 

paradigmatic, 
methodological, analytical, 
inferential, pragmatic, legal, 
ethical, sociopolitical, and 
cultural issues related to the 
application of mixed-methods 
research in psychology 

Describe the application of 1 2 

mixed methods to design; 
data collection, analysis, and 
interpretation; and 
preparation and 
dissemination of research 
findings 

Discuss issues and describe 1 2 

strategies related to 
establishing data quality 
(validity, trustworthiness) of 
mixed-methods research 
findings 

Critically evaluate mixed- 1 2 

methods research studies 


3 

3 


3 

3 


3 


3 


3 


3 


4 5 

4 5 


4 5 

4 5 


4 5 


4 5 


4 5 


4 5 


Produce graphical displays used 1 2 3 4 5 

in descriptive data analysis 
and explain characteristics of 
data in terms of central 
tendency, variability, and 
symmetry 

Explain the assumptions 1 2 3 4 5 

required for parametric 
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hypothesis testing and 
transform a research 
question involving two 
variables into an appropriate 
null and alternative 
hypothesis 

Compute the value, level of 1 

statistical significance, and 
confidence interval around a 
test statistic (including t 
statistic, 2 -score, correlation 
coefficient, U-ratio, and the 
chi-square statistic) and 
interpret the results of 
statistical tests 

Understand the assumptions 1 

and limitations of 
nonparametric statistical 
tests 


Understand statistical concepts 1 

and tests and apply them to 
real-world situations 

Use SPSS to compute statistical 1 

values and perform 
statistical tests 

Use and interpret outputs from 1 

SPSS 

Use SPSS software package to 1 


perform a variety of 
descriptive and inferential 
statistical analyses and to 
interpret their results 

Design, test, and interpret 1 

single-factor, factorial, and 
repeated-measures ANOVA 
designs with interaction 
terms 

Perform appropriate follow-up 1 

tests to test for individual 
level differences within a 
statistically significant 
ANOVA 

Compute post-hoc statistical 1 


2 


2 


2 


2 


2 

2 


2 


2 


2 


3 


3 


3 


3 


3 

3 


3 


3 


3 


4 5 


4 5 


4 5 


4 5 


4 5 

4 5 


4 5 


4 5 


4 5 
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power based on SPSS output 

Design, test, and interpret 1 2 3 4 5 

correlational problems 

Perform statistical and visual 1 2 3 4 5 

data screening and cleaning 

Understand the ways that 1 2 3 4 5 

missing data can be handled 
and the limitations 
associated with these 
methods 

Multivariate analysis of 1 2 3 4 5 

variance and covariance 
(MANOVA / MANCOVA) 

Be able to match statistical 1 2 3 4 5 

questions with specific 
multivariate techniques and 
identify strengths and 
limitations of specific 
analyses 

Use SPSS and theory to screen 1 2 3 4 5 

data and prepare it for 
analysis to include assessing 
assumptions of normality, 
use of procedures that 
account for missing data, 
presence of outliers, and 

linearity _ 
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